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ABSTRACT 

Three types of educational evaluation- -student 
'learning, 'summat'ive, and f ormative--are identified and described. 
Major emphasis is placed upon formative evaluation, .which takes place 
over a period of time and. is performed by professors rather than 
external evaluators or/- judges. Four possible targets for formative 
evaluations are: course content and organization; teacher roles: 
program evaluation; and assessment of special projects oir 
innovations^ Any formative evaluation program should be flexible, 
well directed* and confidentially conducted. The author describes a 
seven-step evaluation system used at McGill University between 1976 
and 1978, and documents an actual case study of. one professor's 
evaluation done in this system. Another formative evaluation system 
recommended for classroom performance techniques is the Teaching 
Improvement Process, developed by the University of Massachusetts. 
Every evaluation system should be subjected to periodic systematic 
evaluations. The key to a flexible formative evaluation system lies * 
with the honestv of professors themselves, and a commitment to 
providing excellent education within a wide range of individual 
styles, subject areas and class types; (DC) 
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. Preface # 
The Standing Committee on Teaching of the Faculty of Education 

t 

at The University of British Columbia has as its goal the promotion 
of quality teaching and teaching improvement within the Faculty. 
Consistent with this goal, we invite leading educators and research and 
development specialists to visit our faculty from. time to time to share their 
expertise and exchange ideas about teaching and teaching improvement. 

We were fortunate that Dr. Patricia A. Cranton of the Centre for 
Learning and Development at McGill University, Montreal, was able to respond . 
affirmatively to 'our invitation 'to vis^it our campus on March 6 and. 7, 1978. 
Dr. Cranton' s activities here included technical consultations on research 
and development with members of the Standing Committee on the Evaluation of 
Teaching (SCET) , the Standing Committee on Teaching (SCOT), and the nascent 
Centre for Improving Teaching and Evaluation (CITE) , advising particularly on 
measurement and computer-based activities of these groups. Additionally, she 
conducted, during the* two days she was here, a large number of personal 

V 

consultations witt^ faculty members concerned with teaching improvement, 
course development, and program evaluation. During the noon hour on 
Monday, March 6, 1978, Dr. Cranton presented a colloquium in the Faculty of 
Education's main auditorium. This colloquium, entitled "The Evaluation of 
Teaching for Improvement 4 -," was attended by faculty members from across the 
campus . 

In response to requests for copies of the text of Dr. Cranton' s 
colloquium, the Standing Committee on Teaching has prepared, with the kind 
assistance and permission of Dr. Cranton, a typescript version. It is our 
hope and intention that making Dr. Cranton's talk available in this format 
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will! facilitate the widespread impact of the points included in her paper. 
Recipients of thist- document having further questions or comments concerning 
its contents are encouraged to communicate directly with t£e author at the 
Centre for Learning and Development at McGill University, or to the sponsors 

of the /document , the Standing Committee on Teaching, Faculty of Education, 

/ * 

The University of British Columbia. If you desire to receive additional copies 

of thil?*paper ,or other paperSi forthcoming' in this series, send your requests 

. ' * f 

to the Standing Committee on Teaching-- 



Stephen F.*wfter and v J . -. Gordon Nelson 

Vancouver, B.C. 
March, 1078 
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EVALUATION OF TEACHING FOR IMPROVEMENT 1 ' 

h Colloquium: March 6> 1978 

The University of British Columbia 

Presented by: Dr. P. A. Ctanton - 
\ Centre for Learning and Development 

McCill University 

I. - What is evaluation ? 

The word "evaluation 11 has strong associations for all of us; we associate 
' evaluation with being "judged/ 1 and consequently, with the possibility of failure. 
Evaluation is basically a judging process --it is usually defined as the pro- 
cedure of collecting information in order to make assessments, judgements or 
decisions. The "failure 11 component of evaluation, however, need not exist. I 
' will define three different types of evaluation,, at least orfe, 6f which does 
not contain the failure component. 

II. T ypes of evaluation in higher education . 

The term evaluation is used in conjunction with three c^ilte .different pro- 
cesses in higher education. First, and perhaps most commonly, we discuss the 
evaluation of student learning. This ^includes the construction of tests for 
student achievement, and the grading of students. Secondly, we are concerned 

with the evaluation of courses, programs v projects and teaching effectiveness 
t 

for the purpose of administrative dec is ion -making. The evaluation, however it 

I ' 
may be conducted, leads to a final decision on the acceptance of a new program,! 

the renewal of^t project funding > the hiring or promotion of staff. This final 

judgement type of evaluation is oftejp referred to as summative. Thirdly, evalua- 

tion is used in the process of improvement > development, or change. Here evalua- 

tion is a cyclical process - no final judgements are made. Rather, evaluation 

is a part of a developmental process, which leads to change which in turn leads 

to further evaluation. It is this third type of evaluation that I will emphasize 

r \ 
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topay, although there will be obvious overlap.; with the other two categories., 
^valuation for improvement is of ton referred ro as formative evaluation. 

"III. Formative evaluation. 

— — — . — _ — — ~ » t 

Formative .evaluation *has several distinctive characteristics. * It is on- 
going, that is, it takes place throughout the duration of a course, or over a 
semester, or over several semesters. The evaluation process is a part of the 

development of the course or of the faculty member's skills as a teacher; it 

* .■ 

is not something done at the end of a course to assess overall effectiveness. 

The professor himself is responsible for the evaluation - he decides what 

aspects of the course or of his teaching performance he is interested in, he 

decides how information will be collected. Although he may consult other people 

or resources* the faculty member* does the evaluation -*no external evaluator 

is entering the scene and making judgements on^the individual's abilities or 

course plans • Because of the extent to which the evaluation procedure is a part 

b'f the course development, the procedure mustNbe i legible enough to accommodate 

many individual differences - differences due to discipline, level of the class, 

size of the class, teacher perso nal ity or style, and dpi on . Since no compari- i 

sons are being made, or rankings being done, difference In procedure become 

$ ■ 

irrelevant * 

There are three or four possible targets for a formative evaluation, 
although again these arbitrary divisions will tend to overlap in practice. The 
evaluation may be centered on the course, i.e., tlie content, organization, 

format, resources, etc* of the cours^ Although it is difficult to separate 

i 

"course" from the "teaching" the division has been a convenient one. There 
are times when one is interested in the teaching -performance or skill in the 
classroom (e.g. , ability to lead discussions* ask questions, speaking ability, 
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.and sc^ on) > Or, one may be interested in the various 'roles a professor plays, 
as teacher, e.g., as evaluator of student learning, as seminar leader, as a 

? 

facilitator of group wcjrk, as a model, as an advisoif, and so on.. In this sense, 
then, we can talk about evaluation of a course and evaluation of teaching as two 
separate targets. The third type of formative evaluation clearly overlaps with 
the others - program evaluation. Here the emphasis is on the integration of . 
courses .into a meaningful unit with goals or objectives of its own. The evalua- 
tion may look at the courses within the program, but the major purpose is to 
examine how each contributes to the program* and to plan possible changes in 
overall organization, phiJLpsophy, or goals* Closely related to program evalua- 
tion, but perhaps a separate category is the evaluation of special projects or 
innovations* As emphasized earlier, formative evaluation is an integral p^r^ of 

the development of the project, a constant monitoring of what is being done. 

» 

For the remainder of the discussion, I will concentrate on^formative evalua- 
te 

tion of teaching effectiveness; however, the distinction between "teaching" and 
"course' 1 evaluation may not necessarily be maintained. 
IVc ^ General description of an evaluation approaptv . ^ 

There ar£ several general concepts or principles which 1 would like to 
point out before discussing specific procedures. I bo.lieve each of these is 
essential foundations for the development of an evaluation procedure. 

First, as mentioned earlier, the evaluation procedures must be flexible. 
No one evaluation process will be equally useful for all individuals; in fact, 
we have rarely ' found, at McGill, two ^individuals who conduct an evaluation in 
precisely the same way., Procedures vary dependent on the subject at^a, the 
level of instruction, the unique characteristics of the students or th$ professor. 
However, this does not mean thatt evaluation is donn in a haphazard or unsystematic 
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fashion - we 'have found it possible to develop a systematic process which has 
in it room for most individual variations* 
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Secondly, since we are interested in evaluation as aft improvement , or 
change process, it is necessary to have available * and linked to the process, 
the resources, materials and personnel for making the improvements'- It is a 
common error that evaluation is embarked upon with no thought as to how changes 
will .be made, or even what changes will be made. Facilities for the change 
process should be. an integral part of the evaluation plan* ' 

I hav^ said that the professor is responsible for doing his own evaluation. 
How does a professor^of history or mathematics know how to conduct an evaluation? 
Several approaches may be used: step-by-sfep written guides may be made. avail- 
able, evaluation consultants or specialists may work with each professor, the 
professor may be "taught 11 about evaluation procedures through workshops, modular 
instruction, or formal or informal class sessions* Regardless of the technlqufe* 
some guidance must be provided • At McGill we use a variety of methods - work-^ 
shops are given to groups of potential users, a User Guide is available and an 
eValuation*consultant is involved, to varying degrees, in each implementation. 

A final basic principle, that may not even need stating, is one of confiden- 
tiality* Evaluation in format! oh is controlled solely by the professor involved - 
he may submit some information to colleagues or department chairman, but it is 
his decision. Case studies, or any information collected by the evaluation con- 
sultant is never released or used in any way without the professor's permission. 
V. An evaluatfon procedure. 

I will describe the steps of the Evaluation System used at McGill Univer- 

i " 

sity. I b*gan"the development of this system nearly two years agoi this yeat 
it has been implemented on a pilot scale, and wil] l.o lAde generally available 
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bv Septeraber. The basic stcvs are applicable to mast types of evaluation done; 
however within this skcleton-f ramework, tremendous variation occurs. 

Step One* The faculty member begins by describing the purpose of the- 
evaluation - why the evaluation is being done, the nature of the course and/or ' 
the teaching situation and the types of changes that, nay be made as a rcsa-.Lt of 
the evaluation. This step fr.cusses the evaluation plan, and encoil^ages thinking 
about the nature of the evaluation to be done - emphasis is placed on questions 
such as "What is thia course intended to accomplish? 11 , "What changes am I willing 
to, or able to make?" Completion of the step may only involve stating pre- 
viously thought-out plans, or may lead to considerable planning and reading in 
the area of evaluation, depending on the individual situation. 

Step Two . The user further specifies frhat will be evaluated by examining 
teaching roles or skills and/or. oourse functions* The prof essor' decides^ what 
particular roles or skills are relevant to him (e*g*, role, of lecturer, seminar 
leader, course manager, etc., or skills of discussion leading, motivating 
students, explaining, organizing, etc*)* Often priorities are established; i.e., 
concentration' on a specific skill where weakness is suspected, or a role that 
-is seen as vital to the particular course. Usually the function of the course 
is also considered at this time - is the course a prerequisite for other courses 
in the area, is it required or electiv^, is it advanced or concerned with basic 
skills,. and so on. ( 

' Step Three > The user moves from the description of the target to the plan- 
ning of the evaluation itself. Sources of information arc selected (students, 
colleagues, professional associations, course notes and* out. lines) and the tech- 
niques for gathering the information are plan: \d (interviews, questionnaires. 

4 

observations, comments, videotapes, etc.)* Specific instruments or questions to 
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be asked are not yet decided, only the strategies that will be employed. Most 

* 

professors consider, at this stage, the strengths and weaknesses of the various 
techniques, and an attempt is usually made to settle on more than one technique 
to compensate for various deficiencies in each, x 

Stop Four . Standards are considered next, before the information is actu- 
ally collected. This is usually done at a fairly general level, asking questions 
suchyas, "What .will t do if 70% of the students respond negatively to this 
question? Or 50% of the students?" A number of variables are considered such 
• as personal or departmental priorities, the relative weights of the various 

issues being examined, the cost of planned changes. It is usually not possible 
or even desirable to attempt to establish definite criteria at this point; 
rather guidelines and priorites are discussed. 

Step Five . i Again as a part of the planning process, the professor looks 
in more detail at the types of changes he is willing and/or able to make. The 
emphasis here is on the available resources tor change and the uioney and time * 
required to jpike changes. We have found that the invest i gat ion^oTVesources 
for change before conducting the evaluation tends to lead to a mpre constructive, 
process. The professor becomes interested in and committed to the concept. of 
improvement or change before information is gathered, evaluation loses its threat 
and often becomes more focussed, and evaluation and improvement become two 
aspects of the same process. Often the error is made of collecting a great 
deal of valuable information, then wondering "What, do 1 do next?" The two pro- 
cesses should not be separate. 

Step Six . The actual information-gathering techniques are selected and/or 
developed. Comprehensive resources should be* available here - the quality of 
the evaluation hinges- on ^his step. All too often a questionnaire is passed 
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f rqa one person to another and used inappropriate* ly simply because people do not 
-'know where to find alternatives* The role of nation consultants may also 

be emphasized at this stage, s 

0 . * 

Step Seven . Finally, the information is gathered, analyzpd and Lnter- 

\ - ^ 

preted. Wh^n the firat ^Jlk steps have been completed carefully, this last J^P 



becomes almost mechanical. I want to emphasize here that this is *\ot the "end 11 

r ^ 

of the evaluation. But the process becomes cyclical. The profe^soV; refers back 
to the resources explored in Step Five and the standards considered in Step * 
Four,. Changes are made, improvement strategies^re implemented , further in forma- ^ 
tion is gathered, and so on. There will \be occasions when one returns to even 



earlier planning stagesi and makes various modifications in the sources of informa- 
tion to be used, the teaching skills to be given priority, etc. 
VI. A case study . * 

I will briefly describe an actual case study of an evaluation doifk with 
this system. The professor involved has given her permission for the evaluation 
to be used as an illustration; however, her name will not be used. 

Prof. Smith approached the Centre for Learning and Development with a re- 
quest for assistance with a course evaluation. She was referred to me and after 
a brief discussion I recommended our Evaluation System to her and set up an 
initial appointment. / 

Prof. Smith is from the Faculty of Education and has been teaching the same 
course for 12 years * She considers herself an innovative and concerned teacher 
and has always attempted to include some form of evaluation (usually a student 
questionnaire) in her course. However, she felt that she was not getting the 
kind of information she needed and was ready to embark on a M more comprehensive 1 ^* 
evaluation. 
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At the first meeting she xv/as introduced to the format of the evaluation 
system, given a User's Guide, and we. began the<diseus>TTon of Step One. Prof. 
Smith decided that she jwas mainly interested in looking at her teaching skills 
and peripherally the content and format of the course. Although she knew her 
^ students were generally satisfied, she wanted to think about possible changes 
or imp lavements, to 'possibly uncover problems she wasn't aware of, and generally, 
to checlo that she wasn't M in a rut" after teaching the, course in the same. way 

for 12 years. She was willing to make changes in her own roles in the class- 

i 

room or to the format of the class sessions, but likely not to the course content. 

She planned to go through Steps 2, 3 and 4 of the process before we met^again. 

At the second meeting Prof. Smith discussed with me the results of hei^y 

planning. She had delineated several teaching roles that were important to her: 

^ activity facilitator, group leader, evaluator of student projects, advisor model 

and discussion leader. More specif ic skillsyof relevance included: asking and 

answering questions, motivating students or providing an atmosphere conducive 

to growth and learning, and meeting the varied needs of individuals in the class. 

She described the course in "detail - a small group of s talents in an elective 

course, with emphasis on activities and group work, student participation and 

involvement. Her major goals for the course were expressed in terms of growth 

and development for the studehts rather than ip terms of knowledge gained* or in 

terms of any absolute criteria,, Three sources of information were selected: 

present students, graduated students and an "outside ^valuator 0 or consultant. 

She planned to use a combination oi interviews v questionnaires; comments, and 

• observations. Standards were discussed in general terms. Given her concern for 

/■ 

individual students in the 'class, Prof. Smith wished to consider all negative 

* 

responses or comments, perhaps exploring the issue with the students if a minority 
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were expressing some pcohU'a. Student opinion i.vmld receive priority over 
information obtained from ttij .outride evaluator. Together we discussed Step ' 
• Five*, ^the exploration of resources and means for making . changes . Prof - Smith 
was already aware of most of th6 university's available resources - I described 

9 * 

some of the Centre's materials, such as a rnqdnfar course on teaching skills, and 

* • * 

our teaching improvement process.*' 

• • ■ 

Step Six, the development of instruments, was attempted by Prof. Smith, 

I 

using a number of the guidelines and resources connected with the Evaluation 
System. Student questionnaires were partly structured (ratings of her ability 
in the various teaching roles) and partly open-ended (questions on student 
growth and development). * Interviews centered on the classroom activities in 
terms of thei?: contribution to student growth and in terms of Prof. Snirth's 
role as facilitator and model. A checklist was *also developed as an aid in the 
classrlfotn observations. These instruments* were discussed during our third 
meeting; "some modifications were made and times were, arranged for interviews 
and observations, . 

-- 

The information gathered from these Y ar i* us techniques showed a generally 
positive attitude toward Prof. Smithes teaching skills and the format of the 
class. Some students felt, however, that more guidance should be provided fo^ 
the in-class activities and that mo£e information in the subject area should be 
given, especially for those students who did not have a ^background in the area. 
A few students asked for more structure in the discussions and more clearly 
defined course goals* Also, two students felt that evaluation based on growth 
was unfair to students starting at a higher leveJ of performance. 

In order to make changes in the course, Prof. Smith decided to improve her 
skills in defining course goals - she felt that t+iis would assist her in dealing 
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with *s eve r^al^ W the issues raised by students* She would us$ objectives to 
.Structure class activities and' discussions . She registered in a modular course 
on tfris topictand also attended a workshop on "contracting," intending to use 



thi^technique to further individualize evaluation of ^tudent progress. Changes 
urrently being implemented and further information co mostly in 
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tfhe form of informal comments and • interviews , to monitor student response to 
the changes. . > 

* Teaching improvement process. 
One alternative to C°r perhaps subset of) the comprehensive evaluation 

. approach is the University of Massachusetts Teaching Improvement- Process ♦ At 
McGill this process is being used in both ways - professors directly contact the 
process or are referred to it .through the evaluation system. 

The Teaching Improvement Process is concetne'd^only with classroom performance , 
although it may overlap with course planning and organization. Twenty teaching 
skills have been delineated which are thought to be relevant across discipline 
and class types. They include such things as:, ability to ask thought provoking 

^^uestions, ability to wrap things up before moving on to a new' topic, ability 
to make clear the relationship between major and minor topics, ability, to inspire 
interest and excitement- in the. course „ and so on. A professor starts the pro- 
cess early in the course* and works closely with a teaching improvement specialist 
throughout the semester. In the third or fourth week of the course, a question- 
naire is given to the students,, asking for ratings on each of the teaching skills. 
The scale used is "needs no improvement 11 to "needs much improvement:" on five 
points. At the same class, the professorVs teaching Is videotaped. The professor 
then meets with the ^aching improvement specialist to discuss questionnaire 
results and view the videotape. One or two problem areas nre selected to be 
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worked on during the semester. Reading materials raay be provided fpr the specific 
ar<*a of concern. The teaching improvement specialist may observe- one or several 
clashes, providing the pro fessor Vith^edbacfc °n his progress- The # videotape 
and questionnaire are repeated towards the end of .the course, using^ly a sub- 
set of the questionnaire Items, concentrating on the area(s) of weakness. 

.As opposed to the varying nature bf the Evaluation System, this jyrocess 
follows a fixed routine for all professors. Generally, faculty response has 
been very positive; however, it has not been systematically 'evaluated. 
VIII. Evaluating the Evaluation System . 

During^ the pilot testing stage, an attempt is being giade to evaluate the 
use of the system - the user guides, the resource materials, the role of con- 
sultants, the process itself, and the effect of the system on the nature of 
evaluations being done and on the types of changes made, A ^discrepancy model" 
(Provost), is being used; i.e., a comparison of what the. system was intended to 
accomplish and what is actually happening. This is being systematically con- 
ducted by a research associate, using interviews with clients "and consultants, 
c^se studies,' and the materials produced by professors for their own evaluations. 

Although the evaluation is not complete, 1 will mention some issues that 
have arisen. One of the most common problems, related to the process and the 
content of the system, is one which we have called the "questionnaire syndrome^! 1 
Very often a professor entering the evaluation system is not prepared for the 
type of planning we suggest*, or for the number ol alternatives available. The 
most common expectation is that they will be able to come in to the Centre and 
leave % hour later with a questi maire to evaluate their course or teaching. 
At times, an initial discussion convinces tl^e professor of the value of a more 
complete evaluation; at other times, he leaves with the questionnaire and is not 
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considered a "system ( user . 11 We are not sure of the extent to which we should 
try to pBrsuade individuals to be more thorough. Also we are not* s-ure of the 

• \ . 

extent to which individuals should be able to use "pieces" of the system; i.e., 
enter and leave at Step Six (developing the evaluation instrument (s) ) . 

A second problem, which I f m sure is' common to all similar processes, is 

the perceived threat of evaluation. This is not as true- for individuals who 

r - ■ r . * ■ 

conie to us for teaching^ oi: course improvement projects; however, we are also 

involved in program and administrative evaluations, where individual professors , 

who did not initiate tjse evaluation, have reservations < How can this fear be 

overcome? Should that be *ur role? When invo^vedNkn a group evaluation, how 

1 do we work with the individual who does not particular] y wish to be involved? 

Should the "majority rule"? Should we act as consultants for a chairman who is 

making evaluation compulsory when one of our principles is that the individual 

professor must agree to and be' responsible for th*^ evaluation procedures and 

ft 

results? • 

V * 

V * 

A third issue of int rest Is the role of the evaluation consultant. We 

perceive our role as being completely advisory v with the professor being respon- 

sible for his own evaluation. However individual expectations vary 4 Often a 

client comes to us expecting that we wj 11 do the evaluation, with his input 

guiding us; i.e., we will develop questionnaire's, do interviews, analyze results^ 

etc. If we are an evaluation s ervice » we should be contributing in these areas. 

We have compromised with some individuals, taking a more* active role; howfever, 

the implications in terms of our^philosophy > time comnii tment and budget are great. 

i 

The 'system was designed to be self-sufficient. Can Jr be;? If not, the format 
and the entire nature of the process would require 1 revision. 

And finally,, an issue that is relevant to all instructional development 
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personnel. How can we best reach the university community? . Our current users 
tend^ to come from the professional schools - from education, medicine, manage- 
ment. Should one of our goals be t to encourage users from the arts or sciences? 
If so, what is the best way to accomplish this? We have worked with individuals 
and groups from history, 'English and chemistry and believe that our .procedures 
are flexible 'enough tg accortfitodate* their needs; however, involvement remains 
low, ~ . 

« 

IX, Summary v 

In summary, I would like to emphasize what I see as the strengths of our 
approach to evaluation for teaching improvement* Tirst » t har^ST'sy^tematic o 
approach is essential, with emphasis placed on the p] Owning of the evaluation. 
Although t;his does require a greater time commitment on Hie part of ■ all individuals 
involved^ the benefits appear to be great* Second, >an evaluation system of any 
kind, must be closely linked with resources for both the evaluation itself and 
the implementation of change, ^valuation 4 should he viewed as one component in 
a larger change process. Third* an evaluation system must he flexible enough 
to accommodate a great variety of individual styles > r.ubject areas, and class 
types. The. key td this .flexibility lies in responsible jty for the evaluation 
being with the professor himself rather than with an evaluator or instructional 
development staff. 
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Footnote * 

Invited colloquium delivered 6 March, 1978, to the Faculty of Education 
* \ 

The University\of British Columbia. Sponsored by: The Standing Com- 

4 I 

lftittee on Teaching (S.C.O.T.)*- Additional copios x of t^iis paper are" 
available from s.C.O.T. 



4 \ 



r 



18 



